Policy relevance of Bayesian statistics overestimated?

OBJECTIVES The observed posterior probability distributions regarding the benefits of surgery for otitis media with effusion (OME) with expected probability distributions, using Bayes' theorem are compared. METHODS Postal questionnaires were used to assess prior and posterior probability distributions among ear-nose-throat (ENT) surgeons in the Netherlands. RESULTS In their prior probability estimates, ENT surgeons were quite optimistic with respect to the effectiveness of tube insertion in the treatment of OME. The trial showed no meaningful benefit of tubes on hearing and language development. Posterior probabilities calculated on the basis of prior probability estimates and trial results differed widely from those, elicited empirically 1 year after completion of the trial and dissemination of the results. CONCLUSIONS ENT surgeons did not adjust their opinion about the benefits of surgical treatment of glue ears to the extent that they should have done according to Bayes' theorem. Users of the results of Bayesian analyses, notably policy-makers, should realize that Bayes' theorem is prescriptive and not necessarily descriptively correct. Health policy decisions should not be based on the untested assumption that health-care professionals use new evidence to adjust their subjective beliefs in a Bayesian manner.